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Bootstrap methods are widely used for distribution estimation, although in some problems 
they are applicable only with difficulty. A case in point is that of estimating the distributions 
of eigenvalue estimators, or of functions of those estimators, when one or more of the true 
eigenvalues are tied. The m-out-of-n bootstrap can be used to deal with problems of this general 
type, but it is very sensitive to the choice of m. In this paper we propose a new approach, where 
a tie diagnostic is used to determine the locations of ties, and parameter estimates are adjusted 
accordingly. Our tie diagnostic is governed by a probability level, /3, which in principle is an 
analogue of m in the m-out-of-n bootstrap. However, the tie-respecting bootstrap (TRB) is 
remarkably robust against the choice of /3. This makes the TRB significantly more attractive 
than the m-out-of-n bootstrap, where the value of m has substantial influence on the final 
result. The TRB can be used very generally; for example, to test hypotheses about, or construct 
confidence regions for, the proportion of variability explained by a set of principal components. 
It is suitable for both finite-dimensional data and functional data. 

Keywords: adaptive inference; bootstrap diagnostic; confidence interval; confidence region; 
functional data analysis; multivariate analysis; percentile bootstrap; principal component 
analysis; spectral decomposition 

1. Introduction 

Bootstrap methods can be particularly effective in distribution estimation, but typically 
only in cases where the distribution being estimated is asymptotically normal. Incon- 
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sistency occurs in many settings, ranging from inference about extremes to problems 
involving goodncss-of-fit testing. 

Arguably the most commonly occurring difficulties of this type arise when estimating 
potentially tied eigenvalues. In this paper we suggest an adaptive, two-stage approach to 
tackling this problem, based on a new bootstrap algorithm. We show that a good bound 
for the distances between eigenvalues and their estimators, founded on an inequality 
borrowed from mathematical analysis, can be combined with the conventional bootstrap 
to give an effective statistical diagnostic for identifying places where ties occur. Armed 
with this information, a new, tie-respecting bootstrap algorithm can be employed to 
generate data that reflect the conclusion of the first bootstrap step. 

Numerical and theoretical properties of the resulting tic-respecting bootstrap (TRB) 
are developed. Together they show that the method can be used reliably in a wide range 
of settings. Our theoretical contributions include a new representation for the limiting 
joint distribution of eigenvalue estimators, valid very generally - for example, in the 
functional data case and in both tied and untied eigenvalue settings. 

A variety of diagnostics can be used in the first stage of the algorithm. The one on which 
we focus is "tuned" using a probability level, /3; the TRB is remarkably robust against the 
choice of this quantity. This contrasts markedly with the m-out-of-n bootstrap, which 
is particularly sensitive to the value of m. We demonstrate this point in a simulation 
study and by proving theoretically that the second stage of the TRB algorithm is largely 
unaffected by the nature of the diagnostic in the first stage; see the first paragraph 
of Section 4.2. We also show that a simple inequality provides a conservative way of 
accommodating the value chosen for /3; see the last paragraph of Section 4.3. 

The TRB is valid in conventional, finite-dimensional settings, where eigenvalues are 
defined in terms of matrices, and also in less standard problems involving functional data 
analysis. Since a non-expert reader may be unable to develop methodology in the func- 
tional data case, we introduce our methodology there. Its vector-case version is entirely 
analogous, and is treated briefly, but specifically, in Section 2.7. Particularly in the case 
of functional data, principal components analysis is a popular way of reducing dimen- 
sion and the sizes of eigenvalues convey a great deal of information about the amount of 
variability that is captured by relatively low-dimensional approximations. 

The problem of bootstrap-based inference for eigenvalues has a history that is not 
much younger than that of the bootstrap itself. As early as 1985, and in the vector case, 
Beran and Srivastava (1985) discussed consistency of bootstrap methods for confidence 
regions, noting that consistency fails in the event of eigenvalue ties. To overcome this diffi- 
culty they suggested avoiding the problem of computing confidence regions for individual 
eigenvalues, and constructing instead a simultaneous region for all the eigenvalues. 

Alemayehu (1988) discussed techniques for finding approximate simultaneous confi- 
dence sets for functions of eigenvalues and eigenvectors. Beran (1988) developed a general 
bootstrap approach to constructing simultaneous confidence regions, and illustrated its 
application using the example of simultaneous regions for eigenvalues. In a general but 
parametric setting, Beran (1997) suggested selecting and adjusting parameter values of 
the distribution from which bootstrap data are drawn, in order to achieve consistency. 
Andrews (2000) developed a theory describing circumstances where bootstrap perfor- 
mance is compromised. 
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Other work on properties of resampling methods for inference about eigenvalues and 
eigenvectors includes that of Nagao (1988), who obtained limiting distributions of jack- 
knife statistics associated with eigenvalue and eigenvector estimation; Eaton and Tyler 
(1991), who introduced techniques for deriving the asymptotic distributions of eigenvalue 
estimators, and illustrated them in the context of the bootstrap; Dumbgcn (1993, 1995), 
who discussed bootstrap-based methods for confidence regions and hypothesis tests re- 
lated to eigenvalues and eigenvectors; Zhang and Boos (1993), who introduced bootstrap 
tests of hypotheses about multivariate covariance structures; and Schott (2006), who 
suggested a test for equality of the smallest eigenvalues of a covariance matrix. 

Hall el al. (1993) described estimation of the largest eigenvalue using the m-out-of-n 
bootstrap in the case of ties. Bickel et al. (1997) and Bickel (2003) also discussed the m- 
out-of-n bootstrap and, in particular, addressed its performance when used to estimate 
distributions that cannot be accessed using the standard bootstrap. However, when used 
in tied-eigenvaluc problems, this technique is uncompetitive with the approach suggested 
here on two grounds: First, it requires empirical choice of m for which a suitable algorithm 
does not seem to be available. Second, it produces distribution estimators that converge 
relatively slowly. 

There is a large amount of literature, too, on principal component analysis for func- 
tional data. In that setting, methodology goes back at least to the work of Besse and 
Ramsay (1986), Ramsay and Dalzell (1991) and Rice and Silverman (1991). The literature 
is surveyed in greater detail by Ramsay and Silverman (2002, 2005). Relatively theoreti- 
cal contributions to the functional-data case include those of Dauxois et al. (1982), Bosq 
(1989, 2000) and Besse (1992). 

2. Methodology 

2.1. Background: conventional estimators of eigenvalues and 
eigenvectors 

Given a random sample X = {Xi, . . . , X n } from the distribution of a random function 
X, let K(u,v) denote the conventional estimator of the covariance function, K(u, v) = 
cov{X(u),X(v)}: 



where X = n~ x Yli-^-i- ^ will be assumed that the argument of X is confined to a 
compact interval T, say, and that u and v are also restricted to that region. 

The eigenvalues 9j and eigenvectors, or cigenfunctions, ipj, arc arguably most clearly 
expressed in terms of the spectral decomposition of the linear operator of which the 
kernel is K: 




(2.1) 



oo 




(2.2) 
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Specifically, denoting the operator too by K, the operator is defined by (Kip)(u) = 
J K(u,v)ip(v) dv, and in these terms, Ktpj = Ojipj- Here and below, unqualified integrals 
are taken over the interval X. 

The estimator K, at (2.1), admits an expansion analogous to that for K, at (2.2): 

oo 

K(u,v)=J20^Au)^(v). (2.3) 

In both (2.2) and (2.3) the eigenvalues are assumed to be ordered as decreasing sequences: 

01 > 2 > ■••>(), 0i > 2 >■••>(). (2.4) 

The inherent positive semi-definitencss of a covariance function guarantees the non- 
negativity claimed in (2.4). The fact that the sample X contains only n elements ensures 
that 9j vanishes for j > n. This in turn implies that the functions tpj in (2.3) are not 
determined for j > n. 

Under mild continuity assumptions on the distribution of the stochastic process X, 
random fluctuations within the data set X guarantee that, even if there are ties among 
non-zero eigenvalues in the sequence 6j, these are not reflected among the empirical 
eigenvalues 9j, with the result that 9i> •••>$„ with probability 1. 



2.2. Principal components 

The principal components of X arc the coefficients = J(X — EX)ipj, and lead to the 
Karhunen-Loeve expansion, 

oo 

X-E(X) = J2^j- (2-5) 

i=i 

The definition of £j implies that those quantities are uncorrected. 

Analogously, the empirical principal components are defined by £y = J(Xi — X)ipj, 
and lead to an empirical version of (2.5), 

oo 

Xi-X = Y,iii$j- (2-6) 

Reflecting the properties E(£j) = and var(^) = 9j enjoyed by the true principal com- 
ponents, we have for their empirical counterparts, 

n i n 

(=1 i=l 

for each j. Since only n data curves Xi are available, then ^ = for j > n + 1 and for 
each 2 . 
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2.3. A tie diagnostic 

If there are ties among the 0j's, then the corresponding values of 9j are generally n" 1 / 2 
apart. Specifically, if 9 p+ i = ■ ■ ■ = 9 p+q , then, under regularity conditions, the differences 
nl ^ 2 {@p+j ~ @p+k) have proper, non-degenerate limiting distributions for 1 < j < k < q. 

Herein lies the difficulty that conventional bootstrap methods have reflecting tied eigen- 
values. In the standard bootstrap approximation to "reality", the 8j's represent the re- 
spective "true" eigenvalues 9j, and so the bootstrap incurs errors of size n^ 1 / 2 when 
it is employed, explicitly or implicitly, to approximate the differences between identical 
eigenvalues. That is, in places where the eigenvalue differences should be zero, their val- 
ues in the bootstrap world are of size n -1 / 2 . This is the same order as the difference 
between an eigenvalue estimator and the true eigenvalue; the extra term of size n -1 / 2 , 
representing a quantity that should really be zero, confounds the distribution-estimation 
problem. In consequence, the bootstrap estimator of the distribution of a tied eigenvalue 
is not consistent. 

We suggest overcoming this problem by, first, using the data to estimate where the 
tied eigenvalues are, and subsequently replacing these empirically-determined ties by tied 
eigenvalue estimators. In principle, estimating the locations of ties requires us to have 
good estimators of the distributions of eigenvalue estimators, and that in turn demands 
knowledge of the locations of ties. We may break this circular argument by using a rela- 
tively robust method for constructing simultaneous confidence bands, such as that given 
below. Depending on the number of ties, and their locations in the eigenvalue sequence, 
our method can be improved by using a more sophisticated approach to constructing 
simultaneous bounds. However, the principle remains the same. 

It is known that, with probability 1, 

sup|^-^|<|||£-iq|, (2.8) 

where, for any bivariate function L, |||L||| 2 = J J L 2 . This property suggests that simulta- 
neous confidence intervals for the 9j 's can be constructed by using a bootstrap procedure 
for estimating the distribution of \\\K — K\\\ . To this end, let x\ , . . . , denote a bootstrap 
resample drawn by sampling randomly, with replacement, from X in the conventional 
way, and put 

1 " 

tft( tt) v) = - £{xt(u) - X\u)}{XJ(v) X\v)}, (2.9) 

i=l 

where X^ = -nT 1 J2i Choosing a probability level /3, such as (3 = 0.05, take Zp to be 
the solution of the equation 

P(\\\lV-K\\\<zp\X) = l-(3. 

Then, approximate and often slightly conservative simultaneous confidence bounds for 
9j are given by 9j ± zp for each j > 1. Properties of this method were explored by Hall 
and Hosseini-Nasab (2006), who showed that the level of conservatism is usually slight. 
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If the confidence intervals (6 j — zp,0j + ip) and (Oj+i — zp,0j+\ + zp) intersect, that 
is, if 6j — 8j + i < 2zp, then our tie diagnostic asserts that Oj = 6j+i. On the other hand, 
if Oj — 9j + i > 2zp, then our diagnostic states that Oj > 6j + i. These conclusions, which 
amount to the results of a sequence of simultaneous hypothesis tests, uniquely define a 
sequence of values of pk and q\ for k > 1 starting with p\ = 0, where pk + 1 and pk + q_k 
define the end-points of the fcth sequence of integers, j , for which the diagnostic asserts 
that the eigenvalues Oj are equal to one another. Note that q^ = 1 if the diagnostic 
suggests that there are no tics for 0p k +i. It follows that 

Pk + <7fc = Pk+i for each k > 1 and 1 = pi + 1 < pi + qi < p 2 + 1 < 

Pi + qi < P3 + 1 < P3 + <?3 < • • • , with the sequence of inequalities (2-10) 

ending when we find a value of v for which q v = oo . 

One may define an alternative tic diagnostic based on the bootstrap distribution of 
su Pj>i \Qj ~ @j\- Let Oj denote the eigenvalues of the covariance operator K\ and zp be 
the solution of the equation 



X =1-/3. 



P [ supW - 6 j\<zp 
\j>i 

The corresponding tic diagnostic states that 9j = 6j + i if Oj — 0j + i < 2zp; Oj > 9j+i, 
otherwise. 



2.4. Adjusting eigenvalue estimators to reflect ties 

Suppose that an empirical tie diagnostic, such as the bootstrap-based method discussed 
in Section 2.3, suggests that ties occur among 0p k +i, ■ ■ ■ , 0p k +q k for k > 1, where pk and 
q\ satisfy (2.10). Then we modify the set of eigenvalue estimators as follows: 

(a) For j in the range pk + 1 < j ' < Pk + Qki we replace Oj by the average, 
Oj say, of the values of 0p k +i, . . . , 0p k +q k , provided 1 < k < v — 1. (b) We 
replace Oj by the average value, Oj, of 0p v+ i, . . . , 9 n if p v + 1 < j < n; and 
we leave the value of Oj unchanged at zero if j > n, but relabel it Oj. 

In addition to producing ties in the estimated eigenvalue sequence when the tie di- 
agnostic says they should be there, part (a) of the algorithm at (2.11) identifies "ties 
of order one", that is, instances where <jk = 1 and the corresponding value of 9j (with 
j ' = Pk + 1 = Pk + <jk ) is equal to 9j . In part (b) we could have replaced 9j by zero when- 
ever j >p v + 1, but that would have altered the total value, °f ^ ne estimated 
eigenvalues. This quantity estimates the total variability of the random function X, and 
is of statistical importance in its own right, without regard to individual eigenvalue esti- 
mators. Therefore, we would prefer to leave it unchanged. 

Implementing the algorithm at (2.11), we generate a new sequence 0\ > 2 > ■ ■ ■ of 
eigenvalue estimators. 
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2.5. Correcting the empirical principal components 

It can be deduced from the second part of (2.7) that, after adjusting the values of the 
eigenvalue estimators 6j to reflect our assessment of where ties lie among the true eigen- 
values, we should also rescale the empirical principal components before resampling. 
Thus, we are led to work with 

iij = $j /Oj ) V2 4 for 1 < i < and 1 < j < n. (2.12) 

These quantities satisfy 

i=l i=l 

Reflecting the uncorrected case, we define ^ = for j > n + 1 and for each i. 

Once we have computed these "corrected" principal components, we resample their 
values in much the same way that we would resample from a set of residuals in a regression 
problem. This is unusual in bootstrap algorithms for functional data analysis; usually 
the raw data are resampled. However, if one were to work instead with the conventional 
principal components £^ , rather than their corrected counterparts ^ , then resampling, 
using the principles outlined below, would be equivalent to resampling from the raw data. 
Although the cigenfunctions can change signs and so can the principal components, this 
does not cause any difficulty since the resampled vectors of the principal components are 
multiplied to the empirical cigenfunctions iftj to produce a bootstrap sample. 



2.6. A tie-respecting bootstrap algorithm 

Let £i = (£ji,£j2) ■ ■ ■) denote the vector of corrected empirical principal components, the 
latter defined at (2.12). Conditional on the data X, draw a resample £*, . . . ,£* by sam- 
pling randomly, with replacement, from the collection £i, . . Put 

oo 

x* = x + X^4, 

this being a bootstrap version of the conventional Karhunen-Loeve expansion at (2.5). 
We shall construct percentile-method bootstrap confidence regions for eigenvalues, using 
the resampled data X* . 

The bootstrap version of K, at (2.1), is given by 

1 n 

K*(u,v) = -J2{X*(u)-X*(u)}{X!(v)-X*(v)}, 
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where X* =n~ 1 Y /l X*. (Thus, K* is the TRB form of K, whereas K\ at (2.9), is the 
conventional bootstrap form.) A bootstrap analogue of the spectral expansion (2.3) is 

oo 

k*{u,v)=Y J e* j r j {u)r j {v). 

The algorithm (2.11) can be applied directly to the bootstrap quantities as at (2.13) 
below, just as it was earlier to the non-bootstrap empirical values. In the bootstrap case 
we employ, for simplicity, the same values j>k and cjk determined by a procedure we gave 
in Section 2.3. 

(a) For j in the range pk + 1 < j <Pk + q_k, we replace 9* by the average, 
9* say, of the values of 9*~ , . . . . §t , - , provided 1 < k < v — 1. (b) We 
replace 9* by the average value, 9*, of 9$ v+ i, ■ ■ ■ , 9* n if p v + 1 < j < n, 
and we leave the value of 9* unchanged at zero if j > n, but relabel it 9*. 

In (2.13) it might be more natural to replace pj and qj by their respective bootstrap 
versions, p* and q*. However, the error incurred by not doing this will generally be small. 

To compute percentilc-bootstrap confidence regions for 6j, we first solve for 
nearly as possible, the equation 

P{9*<9 J +x ja \X)=a. 

Here, a £ (0, 1) represents a probability. Tie-respecting, one- and two-sided percentile- 
method confidence regions, each with nominal coverage 1 — a, are given by 

03 -2j,l-a,°°): {-0O,8j-Xj a ), 0j - %,l-( a /2), &j ~ %a/2)- (2-14) 

A simultaneous confidence region for a general, finite sequence of eigenvalues 9j 1 , . . . , 9j k 
can be constructed analogously. This region will have asymptotically correct coverage, 
even if some of the eigenvalues 9j e are tied with one another or with other eigenvalues 
not included in the sequence. 

TRB confidence regions for functions of the eigenvalues 6*i,#2, ■ • ■ can be constructed 
using the same procedure. We illustrate below in the case of a confidence region for the 
ratio, 




which represents the proportion of the variability of the random function X that is 
explained by the first k principal components. 

Define the tie-respecting estimator p of p, and its bootstrap version p* , by 
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Solve for y a , as nearly as possible, the equation 

P(p* <~P + ij a \X) = a. 

Analogously to (2.14), nominal (1 — a)-level, one- and two-sided percentile-method con- 
fidence regions for p are given by 

(/5-yi- Q ,oo), (-oc, p-y a ), {p-yi-( a /2),p-y a /2)- (2-15) 

In practice, the quantile zp as defined in Section 3 is approximated by the corresponding 
quantile of the "empirical" distribution of ||| A^ — A||| obtained from a finite number of 
bootstrap resamplcs. Different rcsamples yield different approximations of zp, and thus 
produce differing numbers of distinct eigenvalues when using the tie determination rule. 
This may be a source of additional variability in our TRB procedure and may influence 
the coverage performance of the TRB confidence intervals. The additional variability in 
pk and cjk may be significant depending on the value of [3 and spacings of eigenvalues. 
If this were of concern then it could be dealt with by using Brciman's (1996) bagging 
method, even to the extent of deleting simulations that did not accord with the majority 
assessment of the number of eigenvalues. We shall not explore this approach, however. 
One promising aspect of the present TRB method, as found in the numerical study 
presented in Section 3, is that the method is fairly robust against the choice of (3 and 
works pretty well in various settings of spacings of eigenvalues. 

2.7. Adaptation to matrix setting 

If the data Xi are random p- vectors, rather than random functions, then K and A should 
be interpreted aspxp matrices, with A given by: 

1 " 

A=-VpQ-X)(X;-X) T . 
n ^— ' 

i=i 

The eigenvectors ipj are now p- vectors, and infinite expansions, for example, the spectral 
decompositions at (2.2) and (2.3) are now of length only p: 

p p 

Principal components are defined by vector multiplication rather than integration. In 
particular, £^ = {Xi — X) T ipj. Once these reinterpretations are made, the account of the 
TRB algorithm in Section 2.5 is applicable to the case of vector-valued data. 

The L2 norm for bivariate functions L, defined in Section 2.6 by |||A||| 2 = // A 2 , is 
given in the matrix case, where L = (li^), say, by |||A||| 2 = J\ £ 2 iia . With this rein- 
terpretation of notation, (2.8) holds in its original form, and the discussion in Section 2.6 
of both the tic diagnostic and its bootstrap-based implementation is valid in the vector 
case. 
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3. Numerical properties 

Here wc assess the finite sample performance of the proposed TRB algorithm described 
in Section 2.6. The sample functions Xi, i = 1 , . . . , n were generated from the model 

oo 

where X = [—1,1],^- were independently distributed asN(O,0j), &ndtpj(u) = -\/2cos(j7rit). 
Recall that 9j are the eigenvalues of the covariance operator K defined in Section 2.1. 
We set 9j = {500 + 100(j - 4)} _1 for 4 < j < n and 9j = for j > n and considered the 
following three models for the values of B\ , 9 2 and 9$ : 

(1) 01 = 02 = 03 = 1, 

(2) 01 = 1.6,02 = 03 = 0.7, 

(3) 0! = 1.6, 2 = 1,03 =0.4. 

In these models the first three principal components explain most of the variance of X. 
We considered two sample sizes, n = 100 and 400. The proportion of the variability 
explained by the first three principal components equals 99.0% when n = 100 and 98.5% 
when n — 400. 

To compute the estimators 9j, at (2.3), we discretized the sample curves Xi. In par- 
ticular, we took </(< n) equi-spaced points u\,...,Uj on X, and performed a singular- 
value decomposition on the nx J matrix Z = (Xi(uj) — X{uj)). This gave the estimators 
$i > ■ ■ ■ > i) j > of Z T Z, and their corresponding eigenvectors 4>j = (<f>ji, . . . , 4>jj) T ■ The 
eigenvalues 9j and eigenfunctions ?Aj were then obtained from the formulae 9j = / (nJ) 
and ^(sfe) = \/j(f>jk- The principal components = J(Xi — X)t[>j were approximated 
by discretizing the integrals. In our numerical experiments we took J = 30 when n = 100 
and J = 100 for n — 400. In respect to these calculation details, and others given below, 
we used the same settings in the "bootstrap world" as in non-bootstrap cases. 

To assess performance we considered the coverage probabilities of the two-sided confi- 
dence intervals for 9j, at (2.14), and for the ratios pj = J2i<k<j ^/Sfc>i ^fci at (2-15). 
We investigated the performance of the two tie diagnostics described in Section 2.3, one 
based on the bootstrap distribution of \\\K — K\\\ and the other on the bootstrap distribu- 
tion of sup^j \9j — 9j\. In addition to the coverage probabilities, we evaluated how well 
these tie diagnostics identified the tied eigenvalues. Note that for model (1), pt = h + 1 
for k > 2. For model (2), pi~ \ and pk = k for fc > 3, and for model (3), pu = k — 1 
for k > 2. In all cases, p\ =pi = by definition. We computed P(p2 = 3) for model (1), 
P{p2 = 1,P3 = 3) for model (2) and P{f>2 = 1,P3 = 2,p4 = 3) for model (3). These are the 
probabilities of identifying correctly the tics among 9j for 1 < j < 4. Those four terms are 
critical to the success of the tie diagnostics method, since 9j, for j > 4, makes a negligible 
contribution to the total variation of X. 

We also compared our method with m-out-of-n bootstrap algorithms. The latter are 
based on bootstrap rcsamples of size m(< n), drawn by sampling randomly with replace- 
ment from the sample X = {Xi, . . . , X„} in the conventional way. This approach ignores 
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ties among the true eigenvalues. Let X^ = {X.\, . . . ,X^} be the m-out-of-n bootstrap 
resample and be a version of K\ defined in Section 2.3, that is based on X} n , that 
is, 

rn 

Kl{u,v) = -Y J {xKu)-Xl{u)}{x\{v)-Xl{v)}, 
m ^— ' 

i=l 

where XJ n = m -1 Y^i< m Xj . Denote by #J m the eigenvalues of the covariance operator 
and let a m be the solution of the equation 

P(ol n <e J + xl^ m \x) = a. 

Then m-out-of-n percentilc-bootstrap confidence regions can be obtained by replacing 
8j and Xj a in Section 2.6 by 9j and (m/n) 1 / 2 ^ a m , respectively. The normalization 
(m/n) 1 / 2 here derives from the fact that the conditional distribution of m 1 / 2 (^j m — 

9j), given X, approximates the distribution of n 1 / 2 (0 3 — 9j). Results for n = 400 are 
reported in Tables 1-3, which contain coverage probabilities of the confidence regions at 
the nominal level 1 — a = 0.9. 

Table 1. Coverage probabilities of confidence intervals at nominal level 0.9 for the model (1) 



Coverage probabilities for 



Method 






0i 


02 


03 


Pi 


p2 


T 


m-out-of-n 


m/n - 


= 1 


0.818 


0.860 


0.598 


0.670 


0.578 




bootstrap 


m/n - 


= 3/4 


0.830 


0.848 


0.620 


0.702 


0.594 






m/n - 


= 1/2 


0.842 


0.846 


0.620 


0.722 


0.602 






m/n - 


= 1/4 


0.876 


0.854 


0.644 


0.772 


0.636 






m/n - 


= 1/8 


0.892 


0.844 


0.644 


0.792 


0.670 




Tie-respecting 


13 = 


= 0.1 


0.902 


0.902 


0.902 


0.892 


0.892 


1.000 


bootstrap 




0.3 


0.902 


0.902 


0.902 


0.892 


0.892 


1.000 


based on TD1 




0.5 


0.902 


0.902 


0.902 


0.892 


0.892 


1.000 






0.7 


0.902 


0.902 


0.898 


0.886 


0.886 


0.992 






0.9 


0.866 


0.866 


0.848 


0.812 


0.812 


0.910 


Tie-respecting 


P = 


= 0.1 


0.902 


0.902 


0.900 


0.888 


0.888 


0.996 


bootstrap 




0.3 


0.870 


0.870 


0.864 


0.838 


0.838 


0.938 


based on TD2 




0.5 


0.812 


0.820 


0.762 


0.668 


0.668 


0.742 






0.7 


0.764 


0.806 


0.642 


0.448 


0.448 


0.448 






0.9 


0.786 


0.852 


0.576 


0.532 


0.468 


0.138 



Note. Based on 500 pseudo-samples of size n = 400. TD1 stands for the tie-diagnostic based on the 
bootstrap estimate of the distribution of J\K — it'll, and TD2 for the diagnostic based on the bootstrap 
estimate of the distribution of supj >x \9j — 6j\. The numbers in the rightmost column are the values of 
r = P(p2 = 3), the probability of identifying correctly the ties among 9j for 1 < j < 4 in this case. 
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Table 2. Coverage probabilities of confidence intervals at nominal level 0.9 for the model (2) 



Coverage probabilities for 



Method 






6i 


02 


03 


Pi 


p2 


T 


m-out-of-n 


m/n = 


1 


0.864 


0.888 


0.756 


0.882 


0.786 




bootstrap 


m/n — 


3/4 


0.872 


0.888 


0.770 


0.890 


0.776 






m/n = 


1/2 


0.862 


0.876 


0.780 


0.882 


0.762 






m/n = 


1/4 


0.852 


0.874 


0.778 


0.870 


0.770 






m/n = 


1/8 


0.840 


0.876 


0.764 


0.850 


0.758 




Tie-respecting 


P = 


0.1 


0.866 


0.880 


0.880 


0.884 


0.878 


0.994 


bootstrap 




0.3 


0.866 


0.886 


0.886 


0.884 


0.880 


1.000 


based on TD1 




0.5 


0.866 


0.886 


0.886 


0.884 


0.880 


1.000 






0.7 


0.866 


0.886 


0.886 


0.884 


0.880 


1.000 






0.9 


0.866 


0.878 


0.874 


0.884 


0.870 


0.986 


Tie-respecting 


P = 


0.1 


0.866 


0.886 


0.886 


0.884 


0.880 


1.000 


bootstrap 




0.3 


0.866 


0.886 


0.882 


0.884 


0.876 


0.996 


based on TD2 




0.5 


0.864 


0.872 


0.870 


0.884 


0.860 


0.954 






0.7 


0.864 


0.846 


0.794 


0.884 


0.800 


0.810 






0.9 


0.864 


0.860 


0.744 


0.884 


0.776 


0.456 


Note. Based on 


500 pseudo- 


-samples 


of size n 


= 400. The 


meanings of TD1 and TD2 


are the 


same as in 



Table 1. The numbers in the rightmost column are the values of r = P(p2 = 1, p3 = 3), the probability 
of identifying correctly the ties among 8j for 1 < j < 4 in this case. 

Section 2.3 described two tie diagnostics. In the discussion below and in the tables, we 
refer to the tie diagnostic based on the bootstrap distribution of \\\K — K\\\ as TD1, and 
the other as TD2. Let r denote the probability of identifying correctly the ties among 
9j for 1 < j < 4. The results in the tables show that the performance of the m-out-of-n 
bootstrap is sensitive to the choice of m. When ties are present the subsampling scheme 
improves the coverage probability of the conventional bootstrap (m = n), but it suffers 
from the need to choose m carefully. Furthermore, in the case of ties it has poor coverage 
accuracy even when m is chosen to give the best performance. 

In contrast, the TRB method with TD1 diagnostic is quite successful over the entire 
range of (3 in these cases; that is, it is robust against the choice of (3. Moreover, it has 
greater coverage accuracy than the m-out-of-n bootstrap at optimal values of f3 and m, 
respectively. When there are no ties among eigenvalues, the conventional bootstrap works 
well (as expected), but here the TRB methods perform well, too, exhibiting essentially 
the same coverage probabilities as the conventional bootstrap method for f3 > 0.1 in the 
case of TD2, and for (3 > 0.3 in the case of TD1. 

Looking at the probability r one finds that TD1 picks up the tied eigenvalues correctly, 
with very high probability, for all values of (3, while TD2 works better than TD1 for 
small values of (3 when there is no tie. These properties arc directly translated to the 
performance of the methods in terms of coverage probability. Recall that sup^-L \9j — 6j \ < 
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Table 3. Coverage probabilities of confidence intervals at nominal level 0.9 for the model (3) 



Coverage probabilities for 



Method 






6i 


02 


#3 


Pi 


p2 


T 


m-out-of-n 


m/n 


= 1 


0.860 


0.884 


0.876 


0.878 


0.864 




bootstrap 


m/n 


= 3/4 


0.872 


0.872 


0.886 


0.886 


0.854 






m/n 


= 1/2 


0.866 


0.868 


0.884 


0.866 


0.854 






m/n 


= 1/4 


0.838 


0.854 


0.880 


0.826 


0.852 






m/n 


= 1/8 


0.814 


0.828 


0.870 


0.774 


0.830 




Tie-respecting 




= 0.1 


0.742 


0.684 


0.006 


0.736 


0.618 


0.004 


bootstrap 




0.3 


0.846 


0.858 


0.526 


0.868 


0.768 


0.544 


based on TD1 




0.5 


0.858 


0.878 


0.866 


0.876 


0.860 


0.974 






0.7 


0.860 


0.884 


0.876 


0.878 


0.862 


1.000 






0.9 


0.860 


0.884 


0.876 


0.878 


0.862 


1.000 


Tie-respecting 


P 


= 0.1 


0.856 


0.868 


0.584 


0.872 


0.780 


0.634 


bootstrap 




0.3 


0.860 


0.884 


0.876 


0.878 


0.862 


0.998 


based on TD2 




0.5 


0.860 


0.884 


0.876 


0.878 


0.862 


1.000 






0.7 


0.860 


0.884 


0.876 


0.878 


0.862 


1.000 






0.9 


0.860 


0.884 


0.876 


0.878 


0.862 


1.000 



Note. Based on 500 pseudo-samples of size n = 400. The meanings of TD1 and TD2 are the same as 
in Table 1. The numbers in the rightmost column are the values of r = P(p2 = 1, P3 — 2, p4 = 3), the 
probability of identifying correctly the ties among 8j for 1 < j < 4 in this case. 



\\\K - K\\\ at (2.8). This implies that TD1 tends to accept 6j = 9 J+1 more often than TD2, 
for fixed (3. 

In the case n = 100, for which the results are not reported in the tables, we found that 
the TRB does not work as well as when n = 400. Nevertheless, there exists a range of 
(3 for each tie diagnostic such that the resulting TRB does better than the conventional 
bootstrap. One interesting point here is that the TRB recovers quite fast as the sample 
size increases to n = 400. This is true for all cases in the models (1) and (2) where there 
is a tie. For the model (3), one finds that, even for sample size n = 400, the coverage 
probabilities are still far away from their nominal value in a few cases. By increasing the 
sample size further to n — 1000, we found that the coverage probabilities in these cases 
improve greatly. For example, in the case where TD1 with (3 = 0.1 is employed, the cov- 
erage probabilities for 6 1} 9 2 , 9 3l p 1} p 2 equal 0.870,0.872,0.896,0.874,0.892, respectively. 
One lesson learned from this observation is that the coverage error of the TRB converges 
to zero quite fast as n increases, and the convergence depends only a little on the choice 
of (3. 

To see the performance of the TRB algorithm and of the m-out-of-n bootstrap method 
as spacings of eigenvalues change on a continuous scale, we calculated the coverage prob- 
abilities of the two-sided confidence intervals for 0j, 1 < j < 3, when 



(01,02,03) = (l + s,l,l-s), 0<s<0.5, 
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O) — 

d 




spacing 

Figure 1. Coverage probabilities for 62 = 1 as functions of the spacing s = 0% — 61 = 63 — 62 
at nominal level 0.9, based on 500 pseudo-samples of size n = 400. The solid curve corresponds 
to TD1 with P that gives the best coverage performance, and the dashed is for the m-out-of-n 
with the optimal sampling fraction m/n. 



and 9j for j > 4 were the same as those in the three models (l)-(3). Figure 1 depicts the 
coverage probabilities of TD1 and the m-out-of-n bootstrap method as functions of s, 
for which /3 and m/n, respectively, were chosen to give the best coverage performance. 
These results suggest that the proposed TRB method is also robust against spacings of 
eigenvalues. 

Although not reported here, we also implemented bootstrap calibration for each 
method for enhancing coverage accuracy of confidence regions. We found that the double 
bootstrap has little effect on coverage accuracy in the settings considered here. It pro- 
duces confidence regions that have nearly the same coverage error as their non-calibrated 
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versions when the latter do not do well. Calibration slightly improves coverage error, or 
makes it worse, when the coverage accuracy is good. 

The main conclusions from the numerical study are that, in the presence of ties, the 
TRB works well for moderate sample sizes; that TD1 correctly identifies tied eigenvalues 
with high probability, and thus gives good coverage performance for a wide range of 
values of /3 and in various settings of spacings of eigenvalues; and that, in the case of ties, 
this approach outperforms the m-out-of-n bootstrap, in terms of coverage accuracy and 
robustness against choice of tuning parameter. We repeated our experiments for nominal 
level 1 — a = 0.95, and found that the lessons there were essentially the same. 



4. Theoretical properties 

4.1. Distributions of conventional estimators of eigenvalues 

We first describe spacings among eigenvalues using a model that reflects the empirical 
description at (2.10): 

q k > 1 and p k + q k = p k+ i for each k, and 1 = pi + 1 < pi + qi < p 2 + 1 < 
P2 + 12 < P3 + 1 < P3 + 93 < • ■ • , with the sequence of inequalities either 
continuing ad infinitum, in which case we define v = oo, or ending with a , . ^ 
finite value of v for which q v = oo; 9j 1 = 9j 2 for p k + 1 < j\i ji <Pk + Qk 
and for each k; and 9j 1 > 9j 2 if j\ < j2 and j\ and j% lie in distinct intervals 
[p kl + l,p kl + q kl ] and [p k2 +l,p k2 +q k2 }. 

Given a function / of two variables, and functions g\ and 52 of one variable, write 
// /.9i.92 for JJ f(u, v)gi(u)g2{v) dudv. In this notation, and provided that 



sup \K(u, v)\ < 00 and / E(X 4 ) < 00, 

u,v£T J 



(4.2) 



the random variables 



M jlj2 = nl,2 JJ (K ~ K)i> n i> n , j! , j 2 > 1, and 

00 „ 
M = n 1/2 ^((9j -9j) = n 1/2 / (K - K) (u, u) du 



(4.3) 



i=i 



are jointly, asymptotically distributed as multivariate normal variables Nj t j 2 and ■ Njj, 
say, with zero means. In particular, if the principal components £j arc as defined at (2.5), 
and if Zq = J^j ^jj 1 then 
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OO 

cov(N jlh ,Z )=^2 cov (&& . £f )• 
i=i 

Given 1 < fc < v, let J7fc denote the set of indices pk + 1, • • • ,Pk + Qk- Then # 3 = 9^ k \ 
say, not depending on j, for all j ' € J7fe. Theorem 1 below asserts that the limiting joint 
distribution of the normalized eigenvalue differences n 1 / 2 (0 J - — #_,-), and of M , is that 
of random variables Zj, defined as follows: If Jk is a singleton, or cquivalently, if there 
are no ties for the eigenvalue 9 Pk +i, then we take Z Pk+ \ = N Pk+ i, Pk+ i. If Jk contains 
qk > 2 elements, let N^ k > denote the qk x qk matrix with (ji,j2)th component Nj 1 j 2 , for 
Pfc + 1 < j < Pk + qk- Write A Pfc+ i < • • ■ < A P(i+(Zfc for the ordered eigenvalues of N^ k > . Since 
k < v 1 then, with probability 1, these eigenvalues arc also distinct. Define Z Pk+ j = A Pk+ j 
for p k + 1 < j < Pk+qk- 

Theorem 1. Assume {4-1) and (4-&)j an d that 1 < k < v. Then the random variables 
n 1//2 (0j — 6j), for 1 < j <Pk + qk: and Mo, are jointly asymptotically distributed as Zj, 
for l<j<pk+ qk, and Z a . 

4.2. Properties of tie diagnostics 

In Section 2.3 we suggested a particular member of a large class of diagnostics, the general 
class characterized by the property: 

decide that 8j 1 = 9j 2 if and only if \9j 1 — 9j 2 \ < z. (4-5) 

The value of the "critical point" z would generally be computed from data. If £ were to 
satisfy the relations 

0<z = o p (l) and n~ 1/2 = o p (z), (4.6) 

as n — * oo and if the tie diagnostic were given by (4.5), then from Theorem 1 it would 
follow that the tic diagnostic asymptotically correctly identified a finite number of eigen- 
value clusters, that is, 

P(p e = Pi and q e = q e for 1 < t < k) -> 1 (4.7) 

as n — > oo for 1 < k < v. Any sequence z that decreased to zero more slowly than n -1 / 2 
would satisfy (4.6). Theorem 2, below, implies that the sequence z = 22/3, introduced 
in Section 2.3 using a bootstrap argument, satisfies (4.6) if (3 is permitted to decrease 
slowly to with increasing sample size. In Section 3 we show that under (4.1) and (4.2) 
property (4.7) is sufficient for the consistency of TRB. 

To state Theorem 2, let Wj 1 j 2 denote jointly normally distributed random variables 
with zero means and the same covariance structure as ^j 1 £,j 2 — E(^ > j 1 £ i j 2 ). Put 8 2 = 
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T,jEj k (dj - (k) ) 2 , where 6»W denotes the common value of 9j for j £ Jk, and define 

oo oo 

w l= E E K±> ^ 2 =EE^v 

jieJk j2£Jk ji=iji=i 
Recall the definition of _K"f from Section 2.3. 

Theorem 2. issume (^.1) and (^.2), and that \<k<v. Then, (a) E(W) < oo, (h) 
n{@\, . . . , e% \\\K - If f) -> {Wl . . . , Wl W 2 ), (4.8) 

where the convergence is in joint distribution, and (c) conditional on the data X\, . . . , X n , 
we have n\\\K' — K\\\ 2 — » VF 2 m distribution. 

Parts (b) and (c) of Theorem 2 imply that, as n — > oo, the scaled bootstrap critical 
point n 1 / 2 ^ converges in probability to the (1 — /3)-level critical point of W. Also, part (b) 
makes clear in an asymptotic sense the extent of conservatism of the bound 

sup |0,--0 3 -|<|K'-.Kj J (4.9) 

^<j<Pk+1k 

which forms the basis for the tie diagnostic suggested in Section 2.3. To appreciate 
this point, note that the square of the left-hand side of (4.9) is bounded above by 
max(8 2 , . . . , 9 2 ), which in turn is bounded above by J2e<k which, when multiplied 
by n, converges in distribution to Y^e<k^jieJe ^2j?eJe ^iuV This triple series is, with 
probability 1, strictly less than W 2 , which equals the limit, as oo, of n\\\K — K\\\ 2 . 

4.3. Consistency of tie-respecting bootstrap 

The following theorem implies consistency of TRB estimators of the joint distribution of 
any finite number of the 6j's, and of 

Theorem 3. Assume (4-1) end (4-2) and that 1 < k < v. If 'the tie diagnostic algorithm 
used in Section 2.6 to construct the bootstrap algorithm asymptotically correctly identifies 
the eigenvalue clusters up to the kth, that is, if it satisfies {4-7) > then the joint distribu- 
tion of n 1 / 2 (0* — Oj) for 1 < j <pk + qk and n 1 / 2 ^2j(0* ~ 6j), conditional on the data 
X\, . . . , X n , also converges to the joint distribution of Z\, . . . , Z Pk + qk and Zq. 

An immediate corollary is that simultaneous confidence regions for any finite number of 
#j's, and for J2k ® k an< ^ the ratio Y^,k<j ®k/Y^k have asymptotically correct coverage. 
Examples of such confidence regions include those at (2.14) and (2.15). 

We have seen that confidence regions based on our TRB algorithm and the tie diag- 
nostic in Section 2.3 have asymptotically correct coverage if (3 decreases slowly to with 
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increasing sample size. However, the case of fixed (3 is arguably both simpler and more 
easily interpretable. Its treatment makes explicit the cost of uncertainty in the decision 
made at the tie diagnostic stage. To appreciate why, note that Theorem 2 implies that the 
bootstrap critical point izp is asymptotically conservative, in the sense that, for fixed /3, 

lim P ( sup \L - OA < Zb ) > 1 - 13. (4.10) 

The strictness of the above inequality follows from the strictness of the bound 

EE E » J - 

discussed immediately after Theorem 2. From (4.10) it follows that, instead of (4.7), 

liminf P(p e = p e and q e = q e for 1 < £ < k) > 1 - (3. (4-11) 

n — >oo 

Bearing in mind that the limit property in Theorem 3 is conditional on the data, of 
which the quantities pk and qu are functions, it can be seen from (4.11) that a nominal 
(1 — a)-level confidence region based on our bootstrap algorithm, and using the tie diag- 
nostic in Section 2.3 for fixed (3, has asymptotic coverage strictly greater than 1 — a — /3. 
The strictness of this bound provides a degree of assurance of the robustness of our tie 
diagnostic, as we observed in Section 3. 



4.4. Uniformity and perturbations of the model 

In general the bootstrap is not good at producing distribution estimators and confidence 
regions that perform well in a uniform sense. See, for example, Hall and Jing (1995) and 
Romano (2004). Results of Eaton and Tyler (1991) and Diimben (1993) indicate that this 
is also a challenge for 7«-out-of-n bootstrap methods. However, we are not aware of other 
methods that outperform the bootstrap in this respect, and the problem is probably 
inherent, rather than a shortcoming of the bootstrap per se. 

Issues of this type arise if we consider perturbations of the covariance model as n 
increases. To explore this problem, consider the case where we have a triangular array 
of functions, X n> i, X„ t 2, ■ ■ ■ , X n ^ n for n > 1, with covariance kernel K n and eigenvalues 
S n ,i > 6n.2 > ■ ■ • > 0. Suppose that K n — K and 6 n j — 6j converge to zero at rate Si = 
Si(n), denoting a positive sequence decreasing to zero, and that z, in (4.6), converges to 
zero at rate 62- It will be assumed that nT 1 ! 2 is of smaller order than both 8\ and 62- 
If S\ is of strictly larger order than 82, that is, if 82 = o(<Ji), then the probability that 
\@n,ji — @n,j 2 1 > ^ converges to 1 as n — * 00. Equivalently, with probability converging to 1 
the tie diagnostic declares 9j 1 and 8j 2 to be unequal. This is the correct decision in the 
present case, since the rate n -1 / 2 at which the eigenvalues 9 n j 1 and n j 2 are estimated 
by n ji and n ,j 2 i respectively, is of strictly smaller order than <5i, and so the limiting 
distributions of n j 1 and 6*„ J2 are those that arise for unequal eigenvalues. However, if 
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81 is of strictly smaller order than 82, that is, if 61 = 0(62), then the probability that 
\@n,ji ~ 9n.j 2 1 — z converges to 1 as n — > 00, and so with probability converging to 1 the 
tie diagnostic declares 6j 1 and 9j 2 to be equal. This is the incorrect decision, and in this 
instance the TRB generally gives inconsistent estimators of the distributions of n ,h and 

5. Technical arguments 
5.1. Preliminary lemma 

Assume that the kernels, K\ and K2, of two positive semi-definite Hilbert-Schmidt op- 
erators, also denoted by K\ and K 2 , are bounded and admit the spectral decompositions 
K k (u,v) = ^2ij6kj'4 ) kj{ u )'4 ) kj{ v )> where the 9 k j's are eigenvalues and the tpkj' s are the 
respective orthogonal eigenvectors. Suppose, too, that the eigenvalues are arranged in 
order of decreasing size, and that the eigenvalues of K\ can be grouped into blocks of 
length q k , where: 

q k > 1 and p k + q k = p k+1 for each k, and 1 = p\ + 1 < p 1 + qx < p 2 + 1 < 
P2 + Q2 < P3 + 1 < P3 + <Z3 < • ■ • j with the sequence of inequalities either con- 
tinuing ad infinitum or ending with a value of v for which q v = 00; 9\j 1 — 9±j 2 (5.1) 
for p k + l< ji, j2 <Pfc + Ik and each k; and 6\j 1 > 6\ j 2 if j\ < ;}2 and ji and 
j 2 lie in distinct intervals [p kl + l,p kl + q kl ] and [p k2 + l,p k2 + q k2 ]. 

Put || i>\\ 2 =ftjj 2 and n 2 = 11^! - K 2 HI 2 = - ^2 ) 2 ; for r = 1 or 2, define = 

Ep+i<^<p+g "^U I i'U^j and let ip2j denote the projection of ip 2 j onto the space of 
functions spanned by ipi iPk+ i, . . . , ipi, Pk + qk ■ 

Given k > 1, let Jk, be the set of indices j such that p k + 1 < j < p k + q k . Let B denote 
the q k x q k matrix with (ji,j2)th component bj 1 j 2 = JJ(K 2 — Ki)ipij 1 ipij 2 . Then 
B is symmetric, and its eigenvalues, j Pk+ i, . . . ,j Pk+qk , say, are all real, with respective 
orthonormal eigenvectors c Pfc +i , . . . , c Pk + qk satisfying Bcj = jjCj for p k + 1 < j < p k + q k . 
The sum of the squares of the components of B is bounded by 1, and so ||cj£>|| < 1, 
implying that |7j| < 1 for each j. 

Write Cj = (cj >Pk +i, . . . , Cj tPk + qk ) T . Let e k > denote a lower bound to the minimum 
spacing between adjacent 7,-'s; note that e k is assumed strictly positive. Observe that, 
by definition of J k , B X] = 6^, say, not depending on j, for all j £ J k . Order the values 
of 7 Pfc+ i, . . . ,7p fc +</fc as 7( Pfc +i) <•••< l{ Pk +qk)- Lct n denote the permutation of p k + 
l,...,p k + q k that takes p k + j to Tr(p k +j) =Pk + £j, defined such that ^) Pk +i j = J( Pk +j)- 
Ties can be broken arbitrarily; it follows from (5.2), below, that if e k > is kept fixed, 
then ties do not occur if 77 is sufficiently small. A proof of the lemma below is given in a 
longer version of this paper (Hall et al. (2007)). 

Lemma. If (5.1) holds, then for each k > 1 for which q k is finite there exist constants 
CfcSjCfcg > 0, depending only on the eigenvalue sequence 6>n, . . . , ^i,p k +g k +i , on e k > 
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and on \I\ 2 sup \K\\, such that, for r\ < Cks o,nd p^ + 1 < j < pk + qk, 

\92j-0 W -rn*V)\< <W- (5-2) 

5.2. Proofs of Theorems 1 and 3 

Theorem 1 follows from the lemma on taking A i = A and Ki = A, whereas Theorem 3 
follows on letting K\ = A and K2 = K* . In the context of Theorem 1, to treat the joint 
distribution of n x / 2 {6j — 9j) (for any finite number of values of j) and Mo = n 1 / 2 ~ 
9j), we note that the latter quantity has the simple and explicit representation in (4.3), 
which simplifies asymptotically to: 

M ° = / | n_1/2 ^ Xj ^ 2 - jdu + o p (l) 

=E(»- 1/a E(fi«-«4+op(i)- 

3=1 l i=l J 

Therefore, Mj 1 j 2 (for any finite collection of values of (ji, J2)) and Mo, both defined at 
(4.3), are jointly asymptotically distributed as Nj 1 j 2 and i?o; see (4.4). Theorem 3 may 
be treated similarly. 

5.3. Proof of Theorem 2 

Assume, without loss of generality, that E(X) = 0, and define £y = J Xiipj, Vij 1 j 2 = 



1 " 

L(u,v) = - y{X,( u )X 4 ( w ) - #(«,«)}. 



n . 



Then, 



OO CXD „ 

\lL\f=T,J2 U l^ ll|A'-Al-||i||||< I X 2 = O p (n-i). (5.3) 
31=1 ii=i 

Let be as in (2.5). The second part of (4.2) implies that 

„ „ 00 00 

E(AA - A) 2 = E E E (&& - %A^i) 2 < °°- ( 5 - 4 ) 
31=131=1 

Since E(W) equals the double series in (5.4), then (5.4) implies part (a) of Theorem 2. 
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Note that 



E E nE ( u lh)= E E Efe^-^-A) 5 

jl=n+l ]2=r 2 + l ji=r 1 + lj 2 =r 2 + l 

Together, (5.4), (5.5) and the first part of (5.3) imply that for each e > 



(5.5) 



lim lim sup P n 



W - E E ^ 



J 2 



> £ 



(5.6) 



Define {/ = (uj 1 j 2 ) to be a % x g^. matrix, with ji, j'2 G ^7 and Uj U - 2 = n 1 / 2 fj Lipj 1 ijjj 2 = 
n 1 / 2 Uj 1 j 2 . It can be shown from (5.2). on taking K\ = K and Ki = K, that if k < u, 
then the sum of n(6j — 0^) 2 over j G equals the sum of the eigenvalues of U 2 , 
identical to the trace of U 2 , plus a remainder o p (l). Of course, the trace of U 2 equals 



. Therefore, 

E(^-^) 2 =E E^+^O- 



(5.7) 



It can be proved using a conventional central limit theorem that, for each fixed S\, S2 > 1, 
the joint distribution of n 1 ^ 2 Uj 1 j 2 , for 1 < jg < and £ = 1,2, converges weakly to the 
joint distribution of Wj x j 21 for ji,j2 hi the same range. This property, (5.6), (5.7) and 
both parts of (5.3) imply (4.8), and hence also part (b) of Theorem 2. Part (c) follows 
from a bootstrap version of these arguments, which is similar. 
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